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ABSTRACT 



A critical need remains for a practical and low-cost 
methodology to bridge the gap between methodologies for assessment of 
literacy at the national survey level. At the program level, it must be able 
to be used effectively by those in developing countries with limited funds 
but major literacy problems. Etic (external, quantifiable, 
comparison-oriented) measures are clearly important in understanding how 
people acquire literacy, how educators and policymakers view literacy, and 
how economic and societal systems interact in an increasingly interdependent 
world. Collection of international statistics on literacy poses problems that 
specialists have debated. With the greater need for direct measurement and 
increased technical capacity for such measurement in developing countries, 
clarification of issues is occurring in language policy and multilingualism; 
classification; international comparability of data; and measuring the 
consequences of literacy. The Education for All approach that emphasizes 
measuring learning achievement and widening the view to beyond the 3Rs raises 
definition issues. National and local program needs are artificially 
separated by goals and tools typically used in these two domains. Confusion 
exists with regard to the terminology used to describe methods for literacy 
assessment. Challenges to validity of literacy assessment tools are 
cross-cultural comparison, targets and "minimum” levels of achievement, and 
technical/statistical challenges. (Contains 51 references.) ( YLB) 
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INTERNATIONAL LITERACY INSTITUTE 



The International Literacy Institute (ILI), officially co-sponsored by the United Nations Educational, Scientific, and 
Cultural Organization (UNESCO) and the University of Pennsylvania Graduate School of Education, was formally 
established in 1994 in Philadelphia. The ILI builds on more than a dozen years of university-based literacy research, the 
federally funded National Center on Adult Literacy (NCAL), and close collaboration with governmental, non-govern- 
mental, and multilateral agencies worldwide. The ILI and NCAL share the same building with the Penn Technology in 
Education Learning Laboratory (PennTELL) on the campus of the University of Pennsylvania. 

The ILI mission is to provide scientific leadership in training and development in literacy, with a special emphasis 
on developing countries. The ILI defines literacy as primarily the basic skills of reading, writing, and mathematics 
among children, youth, and adults. The term also includes lifelong and work-related skills. 

The ILI has had extensive experience in evaluation and applied research on basic education and literacy in develop- 
ing countries around the world. The ILI’s development activities include partnerships to foster regional and national 
centers of excellence; training to enhance the capacity of national and regional institutions and agencies; innovations 
derived from research, development, and evaluation; information dissemination that provides a forum for the exchange 
of ideas; advanced technologies to increase communication and learning achievement; and training and development 
activities in both formal school settings and nonformal education programs. 



UNITED NATIONS EDUCATIONAL, SCIENTIFIC 
AND CULTURAL ORGANIZATION 



The constitution of the United Nations Educational, Scientific and Cultural Organization (UNESCO) was adopted 
by the London Conference in November 1945, and entered into effect on 4 November 1946 when 20 states had 
deposited instruments of acceptance. It currently has 186 Member States. 

The main objective of UNESCO is to contribute to peace and security in the world by promoting collaboration 
among nations through education, science, culture, and communication in order to further universal respect for justice, 
for the rule of law, and for the human rights and fundamental freedoms which are affirmed for the peoples of the 
world, without distinction of race, sex, language, or religion, by the Charter of the United Nations. 

To fulfill its mandate, UNESCO performs five principal functions: 

• Prospective studies, what forms of education, science, culture, and communication for tomorrow’s world? 

• The advancement, transfer, and sharing of knowledge, relying primarily on research, training, and teaching activities. 

• Standard-setting action', the preparation and adoption of international instruments and statutory recommendations. 

• Expertise', provided to Member States for their development policies and projects in the form of 
“technical co-operation.” 

• Exchange of specialized information. 

(This description is adapted from text on the UNESCO website, found at www.unesco.org.) 
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Abstract 

The World Conference on Education for All (WCEFA) in 1990 at Jomtien, Thailand, included a number of 
educational targets related to out-of-school youth and adults, including (a) to reduce the number of adult illiterates to 
half of the 1990 level by the year 2000 and (b) to improve learning achievement to an agreed percentage of an appro- 
priate age cohort. WCEFA also stressed the need to monitor and evaluate the performance of individual learners as well 
as the delivery mechanisms and outcomes of literacy and other nonformal education programs. At the Mid-Decade 
Review meeting on EFA in Amman (in June 1996; see UNESCO, 1996b), the international community further called 
for efforts at both international and national levels to adopt new techniques and strategies to collect and analyze mean- 
ingful data to monitor progress towards the Jomtien goals. 

Although many countries have been actively striving to meet the basic learning needs of all, current national 
and international capacities remain inadequate for assessing and monitoring the acquisition of literacy and numeracy 
for out-of-school youth and adults. Furthermore, increasing attention to the functional aspects of literacy during the 
1990s has reinforced the necessity to assess individual performance across a continuum of literacy abilities not tied 
directly to school curricula, from the basic 3Rs to the increasingly important area now termed 'basic life skills’ or sur- 
vival skills.’ There remains a critical need for a practical and low-cost methodology that can bridge the gap between 
methodologies for assessment at the national survey level. At the program level the methodology must be able to be 
used effectively by developing countries with limited funds but major literacy problems. 

The present paper provides a critical review of the issues raised above, with detailed discussion of (a) cultural 
dimensions to literacy; (b) international statistics on literacy comparisons; (c) learning, literacy, and the achievement of 
basic and life skills; (d) national level versus local program needs; (e) methodologies for literacy assessment; and (f) 
challenges to the validity of literacy assessment tools. 
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INTRODUCTION 

The Framework for Action adopted by 
the World Conference on Education for All 
(WCEFA) in 1990 at Jomtien, Thailand, 
included a number of educational targets relat- 
ed to out-of-school youth and adults, includ- 
ing (a) to reduce the number of adult illiterates 
to half of the 1990 level by the year 2000 and 
(b) to improve learning achievement to an 
agreed percentage of an appropriate age cohort 
(e.g., 80% of 14 year olds). WCEFA also 
stressed in this regard the need to monitor and 
evaluate the performance of individual learners 
as well as the delivery mechanisms and out- 
comes of literacy and other nonformal educa- 
tion programs. At the Mid-Decade Review 
meeting on EFA in Amman (in June 1996), 
the international community further called for 
efforts at both international and national levels 
to adopt new techniques and strategies to col- 
lect and analyze meaningful data to monitor 
progress towards the Jomtien goals. 

Although many countries have been 
actively striving to meet the basic learning 
needs of all, current national and international 
capacities remain inadequate for assessing and 
monitoring the acquisition of literacy and 
numeracy for out-of-school youth and adults. 
Furthermore, increasing attention to the func- 
tional aspects of literacy during the 1990s has 
reinforced the necessity to assess individual per- 
formance across a continuum of literacy abili- 
ties not tied directly to school curricula, from 
the basic 3Rs to the increasingly important area 
now termed ‘basic life skills’ or ‘survival skills.’ 
There remains a critical need for a practical 
and low-cost methodology that can bridge the 
gap between methodologies for assessment at 
the national survey level. At the program level 
the methodology must be able to be used effec- 
tively by developing countries with limited 
funds but major literacy problems. 

Overall, a reliable and practical information 
base on literacy and basic life skills is required, 
both within and across diverse national and 
cultural contexts. The present paper (as related 
to the Expert Seminar on Literacy Assessment 
of June 22-24, 1998) provides a review of 
issues pertinent to the matters raised above, 
with some proposed suggestions for future 
steps to action in this domain. 



CULTURAL DIMEN- 
SIONS TO LITERACY 
COMPARISONS 

Many countries have attempted to better 
understand national, social, economic, and 
educational policies in light of other nations’ 
successes and failures. In principle, there is 
nothing wrong with comparative analysis; 
indeed comparisons are one of the few ways 
by which nations can gauge their progress. 
However, comparative studies can also be, 
for a variety of reasons, misleading. One 
way to assure that what are termed cross- 
national comparisons are credible is to relate 
them to the long and robust cross-cultural 
and comparative research tradition. Largely 
begun by anthropologists in the colonial era, 
such cross-cultural studies examined and 
compared human behavior ranging from 
child rearing practices and initiation rites to 
the training of craftsmen and sedentary 
agriculturists. Since the 1950s, work on 
international and cross-national educational 
comparisons was undertaken in earnest, as 
exemplified by a broad array of research 
under rubrics such as education and develop- 
ment (Fagerlind & Saha, 1983), cross-cultural 
psychology (Berry, Poortinga, Segall, & Dasen, 
1992), and cultural psychology (Stigler, 
Schweder, & Herdt, 1990). 

Cross-national comparisons of literacy 
acquisition and reading skills are of rather 
recent vintage, beginning with Downing’s 
(1973) seminal Comparative Reading to 
Scribner and Cole’s (1981) well-known 
research on literacy in Liberia, to the more 
recent IEA study of reading in 32 school sys- 
tems undertaken under the auspices of the 
International Association for the Evaluation of 
Educational Achievement (IEA) (Elley, 1992). 
Downing’s work focused on the cultural speci- 
ficities of reading in different languages and 
scripts, Scribner and Cole’s on the cognitive 
consequences of literacy in Africa, while Elley ’s 
study evaluated commonalties of performance 
across a large number of countries from the 
industrialized and developing worlds. 

We know that literacy is not only difficult 
to define in individuals and delimit within 
societies, but it is also charged with emotional 
and political meaning. It was not long ago that 
newspapers and scholars assigned to whole 
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societies a single referent such as “illiterate 
and uncivilized”; and, “illiterate” is still a 
term which carries a negative connotation 
around the world. 

Defining literacy as an individual 
cognitive ability was once thought to be 
simple: it entailed the testing of reading 
and writing skills. This could involve, as in 
American voting laws of the early 20th 
century, the ability to sign one’s name, or 
even mark an “X.” Or, as is done in some 
present-day societies for statistical and 
methodological expediency, literacy may be 
simply inferred from school attendance! those 
with four (or eight or twelve) years of formal 
public schooling are assumed to be literate. 

Or, in yet other societies, literacy rates are cal- 
culated from the numbers of persons who 
answer “yes” to the simple question “Can you 
read and write?” It is now known that such 
approaches to presumed literacy may be quite 
misleading, for a host of reasons. Furthermore, 
when considered as a cultural — as opposed to a 
cognitive — phenomenon, literacy is even less 
well defined, since its meanings, functions, and 
methods of transmission may vary greatly from 
one cultural group to the next. 

The distinction between “emic” and “etic” 
concepts (Berry & Dasen, 1974; Pike, 1966) 
is a central feature in cross-cultural and com- 
parative studies. Emic concepts are those 
that can be understood only within a single 
cultural system or society, and are measured 
only according to criteria relevant to and 
understood within that system. Etic concepts 
are those that are deduced or derived from a 
position outside of any particular system, and 
have as a primary goal the analysis of more 
than one social system or society. 

For example, an etic perspective on literacy 
assumes that skills such as decoding, word-pic- 
ture-matching, and reading a bus schedule 
ought to have substantially the same meaning 
to different individuals and across different 
cultural groups. An emic perspective on litera- 
cy would encompass skills and meanings asso- 
ciated with literacy within cultural groups, 
such as “script recognition” skills in different 
orthographies, as well as the knowledge about 
the values, meanings, uses, and even attitudes 
of/about print in everyday social contexts. 
Simply put, emic literacy skills are those which 
can only be adequately understood within a 
given society; etic skills are those that have 



developed out of the heuristic convenience of 
those who desire a common frame of reference 
or system of measurement. 

The three approaches mentioned earlier — 
Downing, Scribner/Cole, and Elley — provide 
useful examples for examining the problem 
of cultural comparison. Downing (1973) 
asked specialists in a variety of languages to 
analyze the nature of reading development 
within specific languages and orthographies, 
such as Hebrew, Chinese, French, Japanese, 
and English. While most contributors to 
the Downing volume utilized standard frame- 
works for reading analysis, such as word 
decoding and sentence comprehension, they 
also focused on what was specific to the 
nature of learning to read in the language 
and social context of the given society. 

In contrast, Scribner and Cole (1981) 
addressed the classic issue of whether there are 
cognitive consequences to becoming literate, 
through a combined ethnographic and quasi- 
experimental study among the Vai people of 
Liberia. Their methodology focused primarily 
on the cognitive and literacy skills possessed by 
non-literate tribal people, by providing within- 
culture comparisons between schooled and 
non-schooled adults. Among other things, the 
researchers were interested in understanding 
how social practices, such as Quranic study 
and letter writing among commercial traders in 
Liberia, would affect both cognition and litera- 
cy. The results, integrating emic and etic 
approaches to data gathering and analysis, pro- 
vide one of the most in-depth portraits of liter- 
acy and culture that is available to date. 

More recently, the IEA study of reading lit- 
eracy (Elley, 1992) employed an etic approach 
to the study of reading in 32 countries, with 
the general aim of comparing reading achieve- 
ment levels, as well as looking for relationships 
between social variables and cognitive out- 
comes. Considerable efforts were made to 
assure the adequate translation of test items, as 
well as the similarity of the 10- and 14 -year- 
old populations surveyed, but, Elley (1992) 
states, “(i)n the end,... each reader must decide 
for him or herself whether a comparison is 
adequate for a particular purpose," since “com- 
parisons are useful for many purposes and are 
certainly better than alternative comparisons 
made without carefully controlled empirical 
data." While it is difficult to argue against the 
idea of more (empirical) data, it is less than 
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certain that “more” always means “better.” 
Indeed, in the information-driven societies of 
today, there are times when the sheer volume 
of data can overwhelm rational analyses that 
are based on direct observations of local reality. 

In sum, etic (external, quantifiable, com- 
parison-oriented) measures are clearly impor- 
tant in understanding how people acquire 
literacy, how educators and policymakers 
view literacy, and how economic and societal 
systems interact in an increasingly interdepen- 
dent world. Throughout the 20th century, 
literacy has been thought to be associated 
with formal schooling, student achievement, 
and economic success. Thus, it is not surpris- 
ing that quantifiable tools are needed to com- 
pare children with other children, adults with 
other adults, and social and cultural groups 
with other such groups, depending on policy 
and programmatic needs. Nonetheless, the 
psychometric measurement of literacy (espe- 
cially the form taken in large-scale comparative 
surveys) is not the only way to view the acqui- 
sition or possession of literacy skills, though 
it may well be the most convenient and cost- 
effective way for policymakers to measure 
skills across groups of people. 

INTERNATIONAL 
STATISTICS ON 
LITERACY 

In order to provide worldwide statistical 
comparisons, UNESCO has relied almost 
entirely on data provided by its member coun- 
tries (UNESCO, 1983). These countries, in 
turn, typically rely on national censuses, which 
most often determine literacy ability by self- 
assessment questionnaires and/or by means of 
a proxy variable utilizing the number of years 
of primary schooling. Because of doubts about 
the reliability of such data — especially in the 
light of rapid social, economic, and demo- 
graphic changes — considerable concern has 
been expressed about the credibility of literacy 
statistics. In 1986, for example, UNESCO and 
the UN Statistics Office (UNSO) held a joint 
seminar in Paris to discuss the use of house- 
hold surveys to improve adult literacy statistics; 
a technical report which was the basis of this 
seminar was later published (UNSO, 1989). 



While many specialists now agree that exclu- 
sive reliance on indirect measures of literacy 
may be flawed, there is renewed discussion of 
the utility of proxy measures (Murray, 1997). 
More importantly, there have been changes in 
the way that some countries have begun to 
view literacy measurement, especially in North 
American national surveys, and in the OECD 
countries (see discussion of the International 
Adult Literacy Survey, or IALS, study, 1995, 
below). 

The direct measurement of basic skills in 
youth and adults, however, can be technically 
challenging and costly. For these reasons and 
others, most countries (especially in the devel- 
oping world) have not as yet chosen to invest 
in local, national, or regional direct measure- 
ment survey strategies. Furthermore, it must be 
recognized that any change in the methodolo- 
gy used for calculating literacy rates in a popu- 
lation might result in uncomfortable political 
consequences; for example, not long ago in the 
United States the ‘official 1 U.S. literacy rate of 
over 95% was reduced to about 75% of the 
adult population, as a consequence of redefini- 
tion and survey assessment (via the U.S. 
National Adult Literacy Survey [NALS], 

Kirsch et al., 1993). Similar adjustments (in a 
downward direction) were a consequence of 
the IALS study in 1995, and eventually led to 
the withdrawal of France from the final report. 

Overall, the gathering of international sta- 
tistics on literacy poses a variety of problems 
which have been matters of debate among spe- 
cialists. With the greater need for direct mea- 
surement, and increased technical capacity for 
such measurement in developing countries, 
some clarification of issues seems to be taking 
place, as described below. 

Language policy and 
multilingualism 

Most countries have formulated an explicit 
language policy that typically states which lan- 
guage or languages have official status. Often 
the decision on national or official language (s) 
is based on such factors as major or dominant 
linguistic groups, colonial or post-colonial his- 
tory, and the importance of a given language 
to the interests of economic development. 
Official languages are also those most com- 
monly used in primary school, although there 
may be differences between languages used in 
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early schooling and those used later on. 

Further, there may be important differences 
between language policy in primary schooling 
and that of nonformal education (NFE) and 
adult education. For example, in Senegal, 
French is used exclusively in primary school, 
but local Senegalese languages are used in adult 
literacy programs nationwide. The use of 
mother tongue instruction in primary and 
adult education remains a topic of debate 
(Engle, 1975; Wagner, 1992). 

Which languages ‘count’ in literacy statis- 
tics is also controversial. When UNESCO 
gathers literacy statistics, it typically depends 
on government census bureaus to report the 
number of literates and illiterates, as well as age 
and gender differences. Apparently, the lan- 
guage of literacy is not always specified by the 
countries concerned. That is, it would be diffi- 
cult to know, taking the case of Senegal again, 
how many ‘literate’ individuals are literate in 
French, Wolof, Arabic, or other languages, and 
how this is broken down by age and gender. 

While there is usually general agreement 
that all official language (s) ought to be assessed 
in a national literacy survey (e.g., English in 
the United States; English and French in 
Canada, and German, French, Italian, and 
Romanch in Switzerland), there may be dis- 
agreement over the assessment of literacy in 
non-official or semi-official languages, where 
these have a recognized and functional orthog- 
raphy (e.g., Athabaskan in Canada, or 
Hungarian in Romania). In many countries, 
there exist a multitude of local languages that 
have varying relationships and status with 
respect to the official language(s). How these 
languages and literacies may be included in a 
national literacy survey can be a matter of seri- 
ous debate. For example, in certain predomi- 
nantly Muslim countries in sub-Saharan Africa 
(e.g., Senegal or Ghana), the official language 
of literacy might be French or English, while 
Arabic — which is taught in Islamic schools and 
used by a sizable population for certain every- 
day and religious tasks — is usually excluded 
from official literacy censuses. Similarly, litera- 
cy in Chinese, Spanish, Cherokee, and other 
written languages have generally been ignored 
in literacy assessments in the United States. 

Ernie and other perspectives would suggest 
that all languages and scripts should be includ- 
ed in national literacy assessments in order to 
fully understand the range of human resources 



(and to maximize human rights). Practically 
speaking, this poses various problems in fiscal 
and technical resources. However, assuming 
sufficient support for undertaking a household 
literacy survey, effort should be made to inves- 
tigate and assess literacy in all “significant” lan- 
guage populations, where significant might be 
defined by population size or percentage (see 
Wagner, 1990). This is essential especially 
because problems of illiteracy and low literacy 
tend to be over-represented in linguistic 
minority groups. Further, for comparative 
research, studies which include the diversity of 
the languages and literacies within a country 
provide a more accurate measure of social vari- 
ation than those with a more limited focus. 1 

Classification issues 

In the 1960s and 1970s, when many devel- 
oping countries entered the United Nations, 
the majority of the youth and adult popula- 
tions of these countries typically had never 
gone to school nor learned to read and write. It 
was relatively easy in those contexts to define 
such individuals as “illiterate.” The situation 
now is more complex, as some contact with 
primary schooling, nonformal education pro- 
grams, and the mass media is made by the 
majority of families across the globe, and now 
even in the poorest of communities. Thus, 
even though parents in many countries may be 
illiterate or nearly so, it is not unusual for one 
or more of their children to be able to read and 
write to some degree. For this reason alone 
(and there are other reasons, as noted below), it 
would seem that simple dichotomous classifica- 
tions — still in use by many international orga- 
nizations and national governments — ought to 
be replaced by measures that better represent 
the range or continuum of literacy abilities 
which are common to contemporary societies. 

The advent of the WCEFA in Jomtien, 
with its call for an end-of-decade assessment 
in the year 2000, and the growing use of sur- 
veys of learning achievement in the 1990s, 
there is increasing awareness of the utility 
(and challenges) of household survey methods 



1 Nonetheless, It should be noted that it is not 
uncommon for governments to make arbitrary lan- 
guage and literacy “exclusions" from nationally rep- 
resentative samples for reasons of efficacy, politics, 
ethnic strife, and so forth. 
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for measuring literacy levels. In such surveys, 
levels of literacy achievement are ascertained 
based on observed literacy skills. These data 
are then used to perform statistical analyses 
in relation to other variables (such as gender, 
age, schooling, language background, and so 
forth) , as discussed below. 

International 
comparability of data 

The comparability of data — across time 
and countries — is a major concern for 
policymakers and planning agencies. If 
definitions and classifications vary, then it 
can be difficult if not impossible to compare 
data collected through different surveys. 
Comparability and stability are, for example, 
the hallmarks of the UNESCO World 
Education Report and UNESCO’s new World 
Education Indicators project. Nonetheless, if 
comparability is the primary goal, then less 
attention is paid usually to the local and cul- 
tural validity of the definitions and classifica- 
tions of literacy; hence data may become less 
meaningful or potentially misleading. 

International and national needs, defini- 
tions, and research strategies may or may not 
come into conflict over the issue of compara- 
bility, depending on the particular problem 
addressed. For example, as mentioned above, 
UNESCO solicits literacy data worldwide, 
where literacy is measured in terms of the 
number of “literates" and “illiterates." For 
most countries, this dichotomous type of clas- 
sification presents few practical (technical) 
problems and is relatively inexpensive to gather 
(as part of nationwide censuses), while provid- 
ing international agencies with a cross-national 
and time-series framework for analyzing litera- 
cy by geographic or economic world regions. 
However, the fact that this dichotomous litera- 
cy variable may be regarded as simplistic and 
flawed, places serious limits on its usefulness in 
comparative statistics. 

For example, educational planners may 
want to know about the effects of the comple- 
tion of primary or secondary schooling — or of 
a particular literacy campaign — on levels of lit- 
eracy attainment. In these cases, a simple 
dichotomy is a blunt statistical instrument; lit- 
eracy scores or levels are clearly required for 
learning achievement to be adequately mea- 
sured. Furthermore, precise data are needed as 



to which languages and which literacies are 
used in each region and by ethnic group, in 
addition to age and gender variation. 

Compared to census-based literacy esti- 
mates, household-level surveys offer consider- 
able opportunity to create a detailed picture of 
literacy skill profiles and their demographic 
correlates in national populations. As detailed 
in the IALS survey, such surveys can achieve 
considerable cross-national comparability, but 
not without various problems related to item 
comparability (Levine, 1998), and population 
sampling (Kalton et al., 1998). 

The impact on educational policy of such 
comparative studies is due at least in part to 
the realization among national policymakers 
that their national or regional levels of learning 
achievement may be far lower than would have 
been predicted, say, by participation in school. 
The 1994 U.S. National Adult Literacy Survey 
found, for example, that 50% of minority 
group adults in America with a lOth-grade 
education could read at only a 5th-grade level. 
Issues of equity in the United States continue 
to dominate educational policy, as minority 
youth are, on average, about 4 years behind 
achievement levels of same-age White children, 
even up through college years (a Black college 
graduate in the United States reads at about the 
level of a White high school graduate). In a 
recent World Bank national household survey 
in Bangladesh, Greaney (Greaney et al., 1998) 
found that five years of primary schooling 
resulted in only a first grade equivalent of 
learning achievement, and that three years 
of schooling had approximately zero value in 
terms of learning achievement. This study may 
have an important impact on the kinds of 
investments that Bangladesh makes in the areas 
of basic and nonformal education in the future. 

Measuring the 
consequences of literacy 

Will a change from a lower to a higher 
level of literacy skill make a concrete difference 
in an individual’s life? Looking at the “average 
literacy rate” and comparing this statistic 
with health indicators (such as infant mortality 
rates or fertility rates), or estimating “employa- 
bility" from such a rate, cannot adequately 
illuminate the diversity of individual human 
conditions. Nonetheless, the author’s own 
work in rural and low-literate African countries 
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has demonstrated that those with higher litera- 
cy tend to be better off economically (Wagner, 
1993); and the IALS found similar evidence 
across OECD countries. 

Since the 1950s, perhaps the most com- 
pelling argument for human resources develop- 
ment is that literacy and schooling will lead to 
economic growth in countries that make a suf- 
ficient and appropriate investment in them. 
This is the approach sometimes referred to as 
“investment in human capital" 

(Psacharopoulos & Woodhall, 1985). Bowman 
and Anderson (1973), for example, went so far 
as to claim that an 80% national adult literacy 
rate would be necessary for rapid economic 
development, while a 40% literacy rate would 
be required for a minimum take-off of eco- 
nomic development. This type of claim makes 
use of aggregated data across many countries of 
the world, based on a significant correlation 
between gross national product (GNP) and lit- 
eracy rates (using UNESCO data). The notion 
of causality imposed on such correlations is 
very hazardous, but it is still quite common in 
the discourse of economic and literacy plan- 
ners. In one of the few studies of economic 
impact, it was found that, in the agricultural 
sector, an additional year of primary schooling 
directly affected wages and farm output 
(Jamison & Moock, 1984) even though litera- 
cy achievement itself was not measured. 

The issue is not so much whether, but 
rather howto promote literacy in a way that is 
consistent with overall policy objectives. How 
can one most efficiently achieve increased liter- 
acy levels (and important social and economic 
impacts) within the available economic and 
social resources? Here, the emphasis is on effi- 
ciency and appropriateness, domains in which 
specialists — with appropriate measures of 
learning/literacy achievement — can play a cru- 
cial role. What is efficient and appropriate, 
however, brings forth a whole series of quali- 
fiers, such as for whom, in which language, for 
what purpose, using what methods, and so on. 

There is also the issue of program impact, at 
the local level. Various agencies have made seri- 
ous investments in NFE and literacy programs 
in the context of their development strategies 
for women, youth, and disadvantaged adults. 
The measurement of impact of these programs 
not only on learning achievement, but also on 
social and economic outcomes, has become a 
matter of some urgency. The development of 
low-cost measures of assessment in this regard 



might involve not only learning achievement 
but also other indicators (such as attendance, 
attitude, and so forth) that pose special chal- 
lenges, from technical and statistical perspec- 
tives (see below for further discussion on pro- 
gram level assessments). Finally, there is also the 
important question of literacy (or learning) 
retention — how much learning is retained by 
learners for use after program participation ceas- 
es (Wagner, 1998). 

LEARNING, LITERACY, 
AND THE ACHIEVE- 
MENT OF BASIC AND 
LIFE SKILLS 

The WCEFA took a dramatic step by sup- 
porting the notion that basic education con- 
sists of more than conventional (or formal) 
schooling — that education for all should 
include a set of knowledge, skills, and values 
that serve as the basic learning competencies 
(see below). The EFA declaration also made 
clear that learning achievement (rather than 
participation rates, diplomas, etc. that 
predominated in the past) should be the 
primary focus of action. 

In addition, the EFA approach put the 
emphasis not only on measuring learning 
achievement (rather than simply monitoring 
indicators such as expenditures, numbers of 
graduates and teachers, and so forth), but also 
on widening the lens of view to beyond the 
3Rs. Such an approach raises a number of defi- 
nition issues and questions, as discussed below. 

Definition issues 
and questions 

Literacy. 

Many definitions exist for literacy. Two well- 
known and oft-cited definitions are as follows: 

A person is literate who can with 
understanding both read and write a 
short simple statement on his everyday 
life.... A person is functionally literate 
who can engage in all those activities 
in which literacy is required for effec- 
tive functioning of his group and com- 
munity.... (UNESCO, 1978) 
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Using printed and written informa- 
tion to function in society to achieve 
one’s goals and to develop one’s knowl- 
edge and potential. (OECD/Statistics 
Canada, 1995) 

While a comprehensive review of these and 
other definitions cannot be provided here (see 
Venezky et al. , 1990, and Wagner, 1992), there 
is nonetheless substantial consensus that litera- 
cy skills refer to the cognitive abilities of read- 
ing, writing, and calculating. How to measure 
these abilities (and the contents of those mea- 
surement tools) is still widely debated. 
Whatever tools will be used to measure literacy 
will likely have to include measures of these 
three skills. 2 

Learning and life skills. 

Article 1 of the WCEFA Declaration states 
that basic learning needs or competencies 

comprise both essential learning tools 
(such as literacy, oral expression, 
numeracy and problem solving) and 
the basic learning content (such as 
knowledge, skills, values, and atti- 
tudes) required by human beings.... 

(UNESCO, 1990, p. 43) 

Thus, following Jomtien, learning may be seen 
not only in terms of the 3 Rs, but also in terms 
of the kinds of skills, competencies, knowledge, 
and even attitudes relevant to human develop- 
ment. That is, one should focus both on the 
skills and contents of the instructional ‘pack- 
age’- — on what are the purposes of learning 
(and, by extension, of the education provided). 

This definition of basic learning competen- 
cies is extremely broad, and hence poses seri- 
ous problems for those who wish to use it as a 
focus of either planning or assessment. Recent 
attempts to consider learning of ‘life’ or sur- 



2 Calculating — within the broader domain of 
numeracy — while often ignored in the past in 
developing country NFE programs is important to 
include in future work on literacy assessment. But 
including numeracy can be misleading in some 
cases; where applied in NFE programs in develop- 
ing countries, numeracy has unfortunately been 
biased often toward school-based learning, an 
approach which is likely to be less than appropriate 
for those in NFE programs and who use math skills 
in everyday and work activities. 



vival’ skills have received increased considera- 
tion in recent years, but problems of culturally 
appropriate content pose serious concerns. 

One person’s survival skills may be radically 
different from another’s, and these vary not 
only across cultures, but perhaps just as signifi- 
cantly within countries. One illustrative exam- 
ple of this problem is the current development 
of an international life skills survey by the 
OECD, as a follow up to the IALS study. 
‘Technological literacy’ skill is part of this 
prospective study; yet, one can easily imagine 
that poor sectors of developing countries might 
find such a measure to be quite out of place at 
this moment in time. Nonetheless, there have 
been attempts within schooling achievement 
studies to develop measures of life skills (see 
MLA, Chinapah, 1997, discussed below). 

Learning Achievement. 

Learning achievement is understood here 
mainly in terms of what can be measured 
through direct assessment (rather than through 
proxy variables). But it may also be seen in 
terms of who is the end-user, and what are the 
timeline constraints of data gathering; that is, 
what purpose (s) does this measurement of 
learning achievement address? Who will use 
this information, and for what purposes? Some 
end-users (such as service agencies, including 
many NGOs) are uncomfortable with evalua- 
tion and monitoring in general, and with 
learning achievement more specifically. Their 
anxiety is often represented by the agency’s 
focus on service delivered in terms of educa- 
tional participation, rather than impact on 
learning. In other words, learning achievement 
and skill assessment are not neutral topics, and 
may be subject to widely differing approaches, 
which development or national agencies will 
need to choose amongst for their planning and 
programming efforts. 

Clientele (s) . 

Who are the real clients of education and 
development (learning and literacy) programs? 
Beyond the ‘learners’ themselves (children, 
youth, and adults), clienteles might also 
include the following: children (of learners), 
families, communities, political constituencies, 
program staff, government agencies, the pub- 
lic, and others. Are these clients all given the 
same valence, or are some more determinant of 
policy than others, and why? For example, if 
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educational access for the ‘unreached’ (or least- 
reachable) is a policy priority, then the broad 
category of out-of-school youth’ as client may 
require more specification — which youth, what 
gender and ethnicity, what level of poverty and 
distance, and so forth. 



particular due to the inability of large statistical 
analyses to deal with local cultural explana- 
tions. For example, large class size in Japanese 
schools may be less of an impediment to learn- 
ing because of the strong socialization process- 
es that reinforce group solidarity and close-knit 
behavioral norms. 
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Examples of the monitoring 
of learning achievement 

In the domain of learning achievement, 
there are a range of studies — some national, 
others international — that have focused on the 
achievement of reading, math, science, and so 
forth, both in school and out. For example, the 
IEA (International Association for the 
Evaluation of Educational Achievement) 
undertook a number of important internation- 
al comparative studies of learning achievement, 
such as the 1992 Reading/Literacy study 
(Elley, 1992) of 9- and 14- year-olds in 32 
countries (including a number of developing 
countries such as Zimbabwe, Nigeria, and 
Botswana), and the 1996 Third International 
Math/Science Study (TIMSS) of 4th- and 8th- 
grade students in 26 countries (including 
developing countries such as Thailand and 
Iran) (U.S. Department of Education, 1996). 
The 1995 IALS study, in 6 OECD countries, 
measured out-of-school reading, writing, and 
math skills in adults aged 16-65 years. Each of 
these studies contained somewhat parallel 
methodologies for the measurement of learn- 
ing achievement, such as the use of item sam- 
pling pools, translation-retranslation methods, 
and psychometric validation for item rejection. 

In intemational/comparative studies, fac- 
tors such as time-on-task on a particular school 
subject (such as math in the TIMSS) , or the 
impact of TV or book availability (IEA read- 
ing/literacy study) on achievement can be 
studied and disaggregated through multifac- 
toral statistical techniques. In this manner, it 
was found, for example, that the relatively high 
amount of time spent by U.S. students in 
classroom work in math was not a likely cause 
of high math achievement, as several other 
high-scoring countries spent relatively lesser 
amounts of time studying math in class. 
Similarly, it has been shown that smaller class 
size is not strongly determinant of high 
achievement, as some countries (e.g., Japan) 
have large class size and high achievement 
scores. For a number of reasons, such conclu- 
sions have been the subject of serious criticism, 



While there is legitimate interest (and 
sometimes significant amounts of funding) for 
such international comparative studies, there 
has been less interest (and less funding) for 
efforts to measure basic learning achievement 
at the program level. This observation appears 
particularly obvious in LDCs, where NGO- 
based programs rarely have the capacity to 
engage in empirically sound self-evaluation, 
and where international agencies infrequently 
have the resources (human or fiscal) to invest 
in evaluations that include learning achieve- 
ment (there are some well-known exceptions to 
this generalization, such as BRAC in 
Bangladesh orTOSTAN in Senegal). With the 
risk of possible oversimplification, it seems fair 
to say that many (if not most) of the innova- 
tive educational programs sponsored by devel- 
opment agencies in collaboration with NGO 
and/or GO agencies, seldom have the benefit 
of formative or summative evaluations which 
include learning achievement; nor, with few 
exceptions, have such programs invested in 
local capacity building. 

Three projects that have received support 
from UNICEF in the domain of learning 
achievement are worth mentioning in this 
context: the ABC (Assessment of Basic 
Competencies) project in Bangladesh 
(Unicef, 1992), the MLL (Minimum Levels 
of Learning) project in India (NCERT, 1995), 
and the MLA (Monitoring of Learning 
Achievement) UNESCO -Unicef project 
(Chinapah, 1997) in more than a dozen 
LDCs. Each of these projects contains 
interesting dimensions. For example, the 
ABC project represents an attempt to design 
an evaluation methodology that is exceptional- 
ly low in per person assessment cost. The 
MLL project attempts to take a fresh look at 
what kinds of skills can and should be mea- 
sured both in and out of school. The MLA 
project tries to link monitoring methods to 
training and capacity building in diverse 
countries in order to develop new ways of 
promoting a “monitoring culture." 





NATIONAL LEVEL 
VERSUS LOCAL 
PROGRAM NEEDS 

The field of basic education has, over time, 
been subject to a number of artificial separa- 
tions or ‘walls’ between sectors such as formal 
and nonformal, and for children, youth, and 
adults. Bureaucracies and administrative needs 
tend to reinforce such separations, but the con- 
tents of the learning (and assessment) of basic 
skills need not and should not maintain such 
walls. Indeed, it is important to utilize avail- 
able resources across these sectors of activity. In 
a similar way, there has been a tendency, in the 
gathering of literacy data, to think either in 
terms of national level needs or in terms of 
local program (or project-based) needs. Below 
are described the goals and tools typically 
employed in these two domains. 

Goals 

The issues and problems of gathering inter- 
national statistics on literacy fall generally into 
the domain of ‘national level’ data gathering — 
whether through traditional censuses and 
proxy measures, or through household sample 
designs. The general goal of national level data 
collection includes at least the following practi- 
cal purposes: comparison of literacy rates over 
time, determination of statistical relationships 
between literacy and other variables of interest 
(e.g., gender, years of schooling, etc.), and 
cross-national comparability. 

However, there are other major assessment 
needs in literacy work besides those listed 
above. Most of these would fall under the 
rubric of program (or project) impact, plan- 
ning, and development at the local or regional 
level. The specific purposes of assessment and 
evaluation at the program level include the fol- 
lowing: comparison of program effectiveness 
over time; comparison of one type of program 
with others; effectiveness of teacher training; 
and so forth. The emphasis in program assess- 
ment and evaluation is on ‘how programs 
work,’ rather than on national policy planning 
at the national or global levels. At the program 
level, there are real and frequent needs to know 
what kinds of learning achievement impacts 
are being realized. 







Tools 

In the broad domain of nonformal basic 
education, adult education, and literacy educa- 
tion, there has been a tendency to assume 
that the two broad measurement needs — 
national and local/program — would use 
quite different tools or methods for assessment. 
Indeed, this has most often been the case in 
the past, at least in part due to the difficulty 
of finding sufficient technical and fiscal 
resources at the local level to mount impact 
studies. There is no particular reason for this 
situation to remain unaltered. Indeed, there 
are major gains to be made if the methodolo- 
gies and purposes of national and local assess- 
ments were conflated, particularly in the sense 
of shared perspectives, tools, and human and 
technical resources. 

In sum, there is a considerable need to take 
a fresh look at the complementarities between 
national/international and program level infor- 
mation needs, using surveys of direct measure- 
ment of learning, 

METHODOLOGIES 
FOR LITERACY 
ASSESSMENT 

As noted above, there have been many 
efforts to measure literacy achievement over 
past decades. Given the diverse stakeholders in 
the literacy and learning achievement fields, 
there exists a wide variety of tools for assess- 
ment; but there is also some confusion with 
regard to the terminology employed to 
describe such methods. 

Terminology in literacy 
assessment 

A brief list of key terms are as follows, with a 
short summary of how the term is typically used: 

Assessment. 

Assessment refers to one or more methods 
forjudging (via some type of empirical obser- 
vation or test) the actual performance of litera- 
cy or other cognitive skills. For example, the 
UNSO (1989) and IALS (1995) surveys 
included literacy assessment, as did the house- 
hold survey of basic learning achievement in 
Bangladesh (Greaney et al., 1998). Of course, 
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with the wider definition of WCEFA men- 
tioned above, assessment might have to take 
into account broader areas of knowledge, atti- 
tudes, and even values, although consensus on 
assessment tools in these areas remains much 
more problematic (cf. Little & Wolf, 1996). 

Monitoring. 

Monitoring refers to one or more 
methods (sometimes referred to as indicators) 
for gathering and analyzing data related to 
(but not necessarily including) the measure- 
ment of literacy or other cognitive skills. 

For example, monitoring can refer to data 
on rates of schooling or literacy program 
attendance, diplomas attributed, and so 
forth, as included in a wide variety of 
literacy evaluation reports (e.g., UNESCO, 
1996a). The UNESCO-UNICEF project 
on Monitoring Learning Achievement 
(Chinapah, 1997) employed indicators as 
well as tests of learning achievement. 

Survey . 

Surveys typically gather a wide range of 
information, qualitative and/or quantitative, 
and may or may not include performance 
assessments (tests). Literacy surveys, such 
as the UNSO and IALS surveys, gathered 
background information as well as test 
data on individuals. 

Evaluation . 

Evaluation studies (in this case, summative 
evaluation) refer to attempts to judge the 
intended impact of any program of activity. In 
literacy work, evaluations have usually taken 
the form of qualitative or quantitative accounts 
of a given program. In practice, this might 
include the level of learner and teacher partici- 
pation, the attitudes of learners, the number 
who ‘successfully complete the program,’ and 
so forth (see examples in Bhola, 1990; Easton, 
1997). Reliable achievement data has rarely 
been gathered in LDCs in literacy programs. 

Census-based 
literacy surveys 

Since its founding, UNESCO has gathered 
literacy statistics from its member states that 
are almost entirely based on the census surveys 



(mostly on a 10-year [decennial] basis, with 
occasional updates). Given all of the usual lim- 
itations of such surveys in terms of time avail- 
able and competence of census survey staff, it 
is rare that literacy performance measures are 
employed. Thus, most census survey informa- 
tion used to calculate literacy rates employs 
either proxy measures (mainly the highest level 
of schooling reached) or simple questions of 
self-assessment (e.g., “Are you literate?”). Little 
further specification is obtained in such cen- 
sus-based surveys. Hence, such surveys are 
often out of date (given the rapid changes in 
many countries) and very approximative (given 
the low reliability of the data collected). 

Household-based surveys 

Household-based surveys offer considerable 
potential in the field of literacy and basic skills 
learning achievement. This is the case in large 
part due to the many options for flexibility in 
design, such as population sampling, cognitive 
knowledge sampling, and cost management 
(cost can be traded off against gains in statisti- 
cal reliability). For other trade-off issues, see 
Vrignau and Bonora (UNESCO/ILI, 1998) 
and Kolstad (1997). 

For example, with a focus on developing 
countries, Wagner and Srivastava (UNSO, 
1989) produced a relatively low-cost method 
for the development of household surveys 
of literacy, based on a national household 
survey in Zimbabwe. While the analyses of 
the Zimbabwe survey were never fully com- 
pleted, aspects of the methodology were 
adapted by others for household survey use, 
such as in Zieghan (1992) and by the World 
Bank in Ghana (Glewwe, 1991), in Morocco 
(Lavy et al., 1995), and in Bangladesh 
(Greaney et al., 1998). Each of these studies 
utilized a national (as opposed to an interna- 
tional-comparative) framework to produce 
items that could, with reasonable reliability, 
assign individuals to levels of literacy and 
numeracy achievement. Multiple languages 
were used in the UNSO study, with a rudi- 
mentary attempt at psychometric comparabili- 
ty (i.e., more could have done in this regard 
if more resources had been available). 

The IALS survey (OECD/Statistics 
Canada, 1995, 1997; and its predecessors, such 
as the 1993 U.S. National Adult Literacy 
Survey (NALS), invested significant resources 
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in improving the technical and psychometric 
properties of literacy assessment instruments, 
using a variety of techniques, including meth- 
ods for expanding the range of items used in a 
survey format (including especially Item 
Response Theory or IRT). The IALS, and its 
predecessors, utilized a five-level categorization 
method for literacy, along three different scales: 
prose literacy, document literacy, and quantita- 
tive literacy (or numeracy) (for an overview of 
this methodology, seeTuijnman, Kirsch, & 
Wagner, 1997). These survey scales are not 
without critics (e.g., Reder, in press, on the 
NALS and on the colinearity of the three 
scales); and there are criticisms, for example, 
on the degree of international comparability 
(e.g, Kalton et al., 1998, on population sam- 
pling difference across IALS countries) or on 
item comparability (Levine, 1998). 

Such international surveys have also 
been criticized for being too expensive 
and too complicated to be owned’ (that 
is accepted for endogenous and locally 
sustainable use) by national and subnational 
agencies. Nonetheless, these comparative 
surveys have often received considerable 
media and policy attention, and at times 
have led to significant national educational 
policy shifts. The IALS, for example, has led 
the United States, Canada, and a number of 
other OECD countries to invest additional 
resources in adult literacy and adult education 
programs. National or subnational household 
surveys can also have a similar policy impact 
(as in the U.S. NALS), but this result necessi- 
tates a serious and credible study, followed 
by concrete efforts to publicize results. 

Program evaluation studies 

Both formative and summative evaluations 
of literacy programs in developing countries 
are numerous (Bhola, 1990; Carron et al., 
1989). In developing countries especially, 
such evaluations have only rarely included 
psychometrically appropriate tests for measur- 
ing learning achievement. Typically what is 
measured are the inputs in human and infra- 
structural resources, the pedagogical methods 
employed, and the outcomes in terms of atten- 
dance and ‘successful program completion.’ In 
industrialized countries as well, there is insuffi- 
cient attention to the impact of programs on 
learning achievement (Wagner & Venezky, 



1998). This is a very serious gap in the current 
knowledge base. 

Cost issues 

Costs of learning achievement and assess- 
ment studies are quite variable. Estimates of 
the total cost of the IALS survey(s) run as high 
as tens of millions of U.S. dollars, while the 
costs of the UNSO national literacy survey in 
Zimbabwe may be estimated at about US$100 
thousand in current dollars. Costs clearly 
depend on the number of countries included, 
the degree of external expertise required, the 
complexity of study design, and degree of col- 
lection and analyses undertaken. Clear trade- 
offs are available in costing processes, from lim- 
iting sample sizes to the length of tests created 
to the degree of trained personnel required. 
Nonetheless, there is little exactitude in current 
cost estimation due to the paucity of available 
studies with cost figures. 3 (This is also true 
with respect to costs of literacy programs them- 
selves, as described in Wagner, 1995.) 

CHALLENGES TO 
VALIDITY OF LITERACY 
ASSESSMENT TOOLS 

All measures of human behavior are 
subject to what have been termed ‘challenges’ 
to their validity. This is nowhere more the 
case than in literacy education. In part, this 
is historical, in that literacy work has been 
often tied to politics, and thus not subject to 
the ordinary checks and balances of social 
science research and evaluation studies. 
Furthermore, since literacy has been seen over 
time as a skill that can ‘easily be taught’ (for 
example, in ‘each one, teach one’ campaigns), 
it is only relatively recently (the last two 
decades) that the efficacy and efficiency of lit- 
eracy campaigns and programs have come into 
question (and more recently, the same being 
true of formal schooling as well). 



3 According to Greaney & Kellaghan (1996) the 
typical cost components of (national) assessments 
are rougly as follows (as percent of cost): instrument 
development 15; sampling and selection 10; data 
collection 30; data processing 10; data analysis 15; 
reporting and dissemination 15; and governance 5. 
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Cross-cultural comparisons 

If literacy is so deeply tied to the integral 
definition of separate cultures, and if a broad 
definition such as that proposed by the IALS 
is used, then how can common understandings 
of literacy levels or competencies be reached 
across cultures? That is, literacy in poor com- 
munities may be viewed (and, in practice, 
used) in quite different ways than is the case in 
wealthy communities (see Levines critique of 
the IALS, 1998). Indeed, the definition of 
‘poor’ may well have a radically different oper- 
ational definition in different societies, and 
often does. Further, when one attempts to 
compare individuals or groups on whether they 
are self-defined as “literate,” it can emerge that 
different definitions preclude valid compari- 
son. This is often the case with census-derived 
information where large differences in school- 
ing bias the decision matrix for responding to 
the question, “Can you read and write?” 

(cf. Lavy et al., 1995). 

Targets and ‘minimum’ 
levels of achievement 

The WCEFA Framework for Action 
states that: 

Basic learning needs. ..vary.. .from 
country to country depending on the 
actual status of basic education... 
(C)ountries may wish to set their own 
targets, such as a... (r)eduction of the 
adult illiteracy rate.. .to say, one-half its 
1990 level by the year 2000... 
(Furthermore) setting levels of perfor- 
mance and the proportions of partici- 
pants who are expected to reach these 
levels... must be an autonomous task of 
individual countries. (UNESCO, 
1990, Articles 3, 8, and 9, pps. 52-53) 

Since Jomtien, policymakers in many countries 
have given high priority to the improvement of 
literacy and basic education in low-literate 
societies or in low-literate ‘pockets’ of industri- 
alized societies. Yet only a few^countries have 
focused on the minimum levels of basic learn- 
ing competencies implied in the WCEFA 
Framework (see the MLL study in India in 
NCERT, 1995). Further, while a number of 
countries have reinforced their efforts to try to 
reach the ‘one-half’ reduction in illiteracy rates 



called for by the Framework, there is little evi- 
dence that the countries most in need have 
come anywhere near this target, or would have 
the requisite data to support their conclusions. 
Meeting such targets in terms of basic learning 
needs is even more difficult at the national 
level as there is often less consensus as yet on 
what those targets should be. 

There are alternative (and more controver- 
sial) policies for action (beyond simply more 
investment). For example, resources might be 
invested in a more selective fashion (directing 
more funds to preschool and primary schools, 
or to specific groups of adults), so that some 
individuals — those with a greater chance of 
success — would have access to the possibility 
to become literate or more literate. Indeed, 
recent evidence on volunteer literacy efforts in 
the United States suggests that the least literate 
portion of the population is remarkably resis- 
tant to literacy training, often exhibiting much 
higher rates of program attrition and lower 
learning achievement levels (Venezky, 1992). 
Research in developing countries in this 
domain could be very illuminating, and 
might result in quite different conclusions. 

Technical/statistical 

challenges 

There are numerous technical challenges to 
any method of testing. These range from dis- 
putes about theoretical and operational defini- 
tions to the type of statistical tests employed to 
analyze data. Given space limitations, it is use- 
ful to simply note some areas that have been 
particularly debated in studies involving litera- 
cy assessment. These include the following: (a) 
scales of literacy achievement (from dichoto- 
mous, to 4 levels, to many levels); (b) determi- 
nation of when a ‘level’ is achieved (e.g., in 
IALS, it is adequate to say that a level is 
achieved if and only if 80% of the items in a 
level are completed successfully; see Levine, 
1998; Sticht, personal communication, 1996); 
(c) what is included in the operational defini- 
tion of literacy; and (d) effectiveness of the use 
of proxy measures (Lavy et al., 1995; Murray 
et al., 1998; and others). 
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CONCLUSIONS 

In the popular media and in academic cir- 
cles, it is not uncommon to hear that low lev- 
els of literacy and basic skills are among the 
chief problems facing contemporary societies 
today. And, there are numerous arguments that 
would support concern for such a point of 
view, ranging from the economic pressure on 
the workplace, to increases in women’s health 
and in farmer productivity in developing coun- 
tries, and the major problems endemic in 
urban secondary schools in Europe. Literacy 
and basic skills development are part of policy 
discussions in all of these areas, yet needed 
data that would help to empirically address key 
policy issues have often been lacking. 

Improved methods of literacy assessment in 
out-of-school youth and adults can help fill 
this gap and provide new and useful directions 
for future work in this domain. 

Choosing the 
best investment 

It is clear that national and international 
needs may not be one and the same. Countries 
and cultures are diverse, each with a multiplici- 
ty of groups that vary along ethnic, linguistic, 
social class, economic, and other dimensions. 
Each country has its own special history of 
sociopolitical development, and its own experi- 
ences with formal schooling and broader edu- 
cational development. The international policy 
community has its interests as well, mostly in 
trying to guide national decision-making from 
indices of where nations “fall” on some scale of 
economic productivity or worker efficiency. 
Thus, the improvement of literacy assessment 
in a comparative context may affect local, 
national, and international interests in con- 
trasting ways. National interests and “internal” 
considerations (involving, for example, popula- 
tion diversity) may be seen as nettlesome prob- 
lems or simply constraints by planners con- 
cerned with international comparison. On the 
other hand, national considerations about pop- 
ulation diversity, linguistic variations, and even 
orthographic diversity (such as unusual features 
of a script) may have to be sacrificed in order 
to achieve a larger basis for international com- 
parison. For these and other reasons, there is 
ample logic for local programs and national- 
level policymakers to hesitate in supplanting 
local concerns and interests for those of the 
regional or international comparisons. 



More specifically, the level of resource 
investment in empirical data gathering in IEA- 
like studies is, for many developing countries, 
far greater than that made heretofore. Thus, 
there may be opportunities to create a research 
infrastructure through the carrying out of 
international comparisons, not so different in 
kind from that achieved by anthropologists 
working with diverse groups in developing 
countries. Perhaps most importantly, compara- 
tive studies can, if properly designed, help to 
achieve a greater understanding of cultural val- 
ues and attitudes towards learning and literacy. 
Such an approach would enable the problem 
of diversity to be turned into an opportunity 
of studying varieties of learning in context. 

What then can be taken from “horse race" 
comparisons (implicit in IEA- or IALS-like 
studies), where one country edges out another 
by a few questions on a test? It would seem 
that those comparisons are of less immediate 
interest than what can be learned from cross- 
cultural and within-culture variations. 
Unfortunately, sponsors (i.e., funding agencies) 
of comparative studies usually need the com- 
petitive (horse race) aspects to ‘sell’ the interna- 
tional study to policymakers. Hopefully, a bet- 
ter understanding of the opportunities embed- 
ded within cross-national comparisons can lead 
to a useful change in this funding discourse. 

New directions in literacy 
assessments: Asking the 
right questions 

To undertake program (local) as well as 
national and international assessments 
requires an unusual degree of sensitivity to 
sociocultural variation as well as to the com- 
plexities of psychometric and evaluation 
techniques. The above discussion necessarily 
raises questions that will need to be addressed 
in order for serious progress to be made in 
the domain of assessment of literacy and 
learning achievement. Certain questions enu- 
merated below have received attention over 
the years, but remain without broadbased con- 
sensus for the moment; others are relatively 
new in policy discussions. 
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Determining basic 
learning skills. 

What is contained in the set of basic learn- 
ing skills said to be commonly needed in basic 
education programs in developing countries? 
Beyond the 3 Rs, little consensus has as yet 
been achieved on this question, even though 
the WCEFA and many of those working in the 
field seem to have a sense of what ‘basics’ are 
needed. While examples of how to engage the 
process of developing core (basic skills) compe- 
tencies exist (see earlier discussion of MLA 
project), they do not provide sufficient clarity 
as to how a core set of life skills can be broadly 
applied or agreed upon, especially in out-of- 
school settings. In other words, what types of 
skills must be acquired in order to guarantee 
‘basic learning needs’? This question requires 
greater conceptual consensus, but also empiri- 
cal testing beyond the rhetoric still endemic in 
various policy discussions. 

Learning at the 
program level. 

How many educational (and social/eco- 
nomic) development programs actually mea- 
sure the quality of the program in terms of 
learning achievement and other related vari- 
ables? Using what methodologies, and over 
what time period? For example, in a health 
education program concerning HIV for 
young women, what types of knowledge, 
skills, or behavioral changes would constitute 
adequate learning achievement? How could 
these measures be linked to national level a 
nd household survey data collection? One 
useful direction would be to have agencies 
collect an inventory of learning that is expect- 
ed within programs whose main emphasis is 
not strictly educational (but includes signifi- 
cant learning components). 

‘Shareability’ of 
assessment tools. 

How can both comparability and context 
sensitivity be appropriately balanced in literacy 
and basic skills assessments? Can the tools 
developed for household surveys be ‘shared’ 
with program evaluation and vice versa? What 
kinds of savings can be made in human and 
fiscal resources across these two domains? The 
notion of shareability may be one way to find 
a compromise between etic approaches to strict 



(international) comparability on the one hand, 
and emic approaches to strict national, subna- 
tional, and local contextuality on the other. 

Timeliness. 

Until recently, national literacy data could 
only be drawn from decennial censuses. More 
timely data is clearly needed, given the rapidly 
changing economies and societies in every 
region of the world. A major question is how 
can a model of ‘continuous’ assessment be put 
into place for both the program and national 
levels? Are there methods for undertaking 
reliable household sample surveys that are 
cheaper, more accurate, and more timely 
than census data? 

Where does learning happen? 

Within a learning achievement perspective, 
the oft-cited distinction between formal and 
nonformal education may be seen to be of 
declining importance. Similar knowledge 
and skills can be instructed and learned 
whether in or out of schools, whether with 
children, youth, or adults. If the focus is on 
learning, then at least one goal would be to 
better understand which contexts for learning 
provide optimal access, quality, and relevance. 
In other words, where do children, youth, 
adults, and families most effectively learn 
what they need to learn? 

Who does literacy 
assessment, and the 
problem of capacity building. 

Assessments necessarily require a variety of 
expertise. The providers of this expertise 
include policymakers, psychometricians 
(test makers), survey and interview specialists 
(enumerators), and data analysts, as well as 
learners and instructors for preparatory pilot 
testing. Capacity building — the training of 
sufficient expert staff — has been a major 
impediment in the past. How such resources 
can be put into place in the future will be a 
major question to address, at local, national, 
regional, and international levels. 




20 



New directions for action: 
Some next steps 

Based on such questions, agencies con- 
cerned with literacy assessment (and basic 
learning needs) will likely have to consider a 
number of steps in the near and medium term 
in order to develop a proposed action plan in 
this domain: 

• Initiate a series of several national 
pilot studies on literacy assessment 
and basic learning achievement, 
employing methodologies of use 
across the program- and national- 
level needs. In the form of household 
surveys, these studies would entail 
program-specific (and learner-specif- 
ic) observations of what types of 
learning take place in school, and out 
of school, as well as in health, agricul- 
ture, and the like. 

• Develop low-cost and quick meth- 
ods for the measurement of literacy 
and basic learning achievement that 
might be applied to a broad range of 
education and development activities, 
and that take advantage of the notion 
of ‘shareability’ between both national- 
and program-level data needs. 



about learning achievement can alter (and 
improve) policies and practices at all levels of 
intervention. 

By better understanding and avoiding the 
simplicities of the past, more progress can be 
made in the future. There still appear to be 
many educational development projects with 
learning components in which the impact 
seems to remain primarily based on conjecture 
or optimism. This situation needs to change in 
development work, but how it will change, 
and what roles different agencies will play, 
remain less certain. 

None of the above activities will be success- 
ful without the effective integration of local 
and national capacity building, which needs to 
be an integral part of any future effort in this 
regard. The resulting plan of action should be 
able to enhance not only national-level assess- 
ments but also provide important impact data 
for programs seeking self-improvement. In this 
manner, national and international agencies in 
literacy and basic education will have more 
effective tools at hand for this central domain 
of human development. I 



• Find ways within development 
and national agency activities to 
link learning achievement in educa- 
tion and other sectors (e.g., health, 
nutrition, agriculture, work, etc.) 
with the measurement of basic 
learning competencies. 



• Develop a near-, medium-, and 
long-term plan for capacity building 
in the area of literacy and learning 
assessment. Training needs to be 
reinforced at all levels where reliable 
data is expected to be collected, 
analyzed, and reported. 



Final thoughts 

The task of rethinking literacy assessment 
will require a better delineation of what litera- 
cy and basic (and life) skills mean, how they 
can be measured, and how knowing more 
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